Recent advances in computing power and related technology have fostered the development of a new generation of powerful software applications. Gaming applications, communications applications, and multimedia applications have particularly benefited from increased processing power and clocking speeds. Indeed, once the province of dedicated, specialty workstations, many personal computing systems now have the capacity to receive, process and render multimedia objects (e.g., audio and video content). While the ability to display is (receive, process and render) multimedia content has been around for a while, the ability for a standard computing system to support true multimedia editing applications is relatively new.
In an effort to satisfy this need, Microsoft Corporation introduced an innovative development system supporting advanced user-defined multimedia editing functions. An example of this architecture is described in U.S. Pat. No. 5,913,038, issued to Griffiths and commonly owned by the assignee of this document, the disclosure of which is expressly incorporated herein by reference.
In the '038 patent, Griffiths introduced an application program interface which, when exposed to higher-level development applications, enables a user to graphically construct a multimedia processing project by piecing together a collection of “filters” exposed by the interface. The interface described therein is referred to as a filter graph manager. The filter graph manager controls the data structure of the filter graph and the way that data moves through the filter graph. The filter graph manager provides a set of object model interfaces for communication between a filter graph and its application. Filters of a filter graph architecture implement one or more interfaces, each of which contains a predefined set of functions, called methods. Methods are called by an application program or other objects in order to communicate with the object exposing the interface. The application program can also call methods or interfaces exposed by the filter graph manager object.
Filter graphs work with data representing a variety of media (or non-media) data types, each type characterized by a data stream that is processed by the filter components comprising the filter graph. A filter positioned closer to the source of the data is referred to as an upstream filter, while those further down the processing chain is referred to as a downstream filter. For each data stream that the filter handles it exposes at least one virtual pin (i.e., distinguished from a physical pin such as one might find on an integrated circuit). A virtual pin can be implemented as an object that represents a point of connection for a unidirectional data stream on a filter. Input pins represent inputs and accept data into the filter, while output pins represent outputs and provide data to other filters. Each of the filters includes at least one memory buffer, and communication of the media stream between filters is often accomplished by a series of “copy” operations from one filter to another.
A filter graph can have a number of different types of filters, examples of which include source filters, transform filters, and render filters. A source filter is used to load data from some source, a transform filter processes and passes data, and a render filter renders data to a hardware device or other locations (e.g., to a file, etc.).
FIG. 1 shows an exemplary filter graph 100 for rendering media content. Filter graph 100 comprises a number of different filters 104-110 and may or may not comprise a source 102. A typical filter graph for multimedia content can include, for example, of graph portion that is dedicated to processing video content and a graph portion that is dedicated to processing audio content. For example, in FIG. 1 a source 102 provides content that is typically in compressed form. A source filter 104 receives the content and then provides the content to one or more decoder filters for decompression. In this example, consider that filters 106-110 process video content and filters 106a-110a process audio content. Accordingly, the decoder filters decompress the data and provide the data to a transform filter (e.g. filters 108, 108a) that operates on the data in some way. The transform filters then provide the transformed data to a corresponding render filter (e.g. 110, 110a) that then renders the data.
Typically, an application program or application 112 provides a means by which a user can interact with the content that is processed by the filter graph. Responsive to a user interacting with the application, the application can issue commands to the source filter 104. Examples of commands can include Run, Stop, Fast Forward, Rewind, Jump and the like. The source filter receives the commands and then takes steps to ensure that the commands are executed at the right time. For example, the source filter 104 typically receives data and provides timestamps onto data samples that define when the data sample is to be rendered by the render filters. The source filter then hands the timestamped data sample off to the decoder for further processing. The render filters now know, because of the timestamp, when the data sample is to be rendered.
It can be desirable, in situations such as this, for the application to be informed when a particular data sample has been rendered, or when an event associated with a particular data sample has actually occurred. This is because the application may have other tasks that need to be accomplished and which cannot a be accomplished until certain events have occurred. Events can include the start of the stream, end of the stream, changes in the stream's content, specific points of interest and the like. Changes in the presentation rate, such as the playback speed or direction, can further complicate the situation.
Presently, the way that the application is notified when particular events occur is that the render filter 110 is notified that the application wishes to be informed when a particular event occurs. The render filter waits until the particular event actually occurs (e.g. a data sample is rendered), and then sends a notification to its upstream filter—in this case the transform filter 108. The transform filter 108, in turn, sends the notification to the decoder filter 106 which then relays the notification to the source filter 104 and then onto the application.
Relying on the render filters to notify the application increases the design complexity of the output components and their workload responsibilities. Additional problems with this approach include that if the application requests a notification when a particular event has transpired, then the request itself must be either sent as a separate piece of data along with details indicating when the message should be sent, or the request must be attached to the data to be displayed. Additionally, a delay accumulates as the request propagates down the filter graph and as the notification returns from the filter. The delay can also be affected by process scheduling or network conditions and is beyond the control of the application.
Another approach to notifying the application when a particular event has transpired can be to establish a direct connection between the application and the render filters. This, however, is undesirable because it necessarily binds the render filter to the application and makes re-using the render filter for different applications impossible since each filter must have complete knowledge of the other. Different render filters cannot then be used with the same application.
Products utilizing the filter graph have been well received in the market as it has opened the door to multimedia editing using otherwise standard computing systems. It is to be appreciated, however, that the construction and implementation of the filter graphs are computationally intensive and expensive in terms of memory usage. Even the most simple of filter graphs requires and abundance of memory to facilitate the copy operations required to move data between filters. Complex filter graphs can become unwieldy, due in part to the linear nature of filter graph architecture. Moreover, it is to be appreciated that the filter graphs themselves consume memory resources, thereby compounding the issue introduced above.
Accordingly, this invention arose out of concerns associated with providing improved methods and systems for processing media content. In particular, the invention arose out of concerns associated with providing improved filter graph systems and methods.